fix(mongodb): store SET values as sent in the numeric ADD pipeline update - #589

Fixes #573.

On MongoDB, an UpdateItem that SETs a value and also ADDs a number could store something other than the value sent. A $ string was read as an expression and a map merged into the map it replaced, and several of these left an item every read answered with InternalServerError. The value is now stored as sent.

What happened on main

An unconditional UpdateItem that asks for no old values and no consumed capacity, on a table with no index and no stream, is built as a native MongoDB update while the server records that the table has no index. The first write on the session-scoped read-modify-write path that finds no such record, or one older than 60 seconds, reads the index catalog and records the answer; no write renews the record during its 60 seconds. With a numeric ADD, try_build_native_update in crates/storage-mongodb/src/data_engine.rs builds an aggregation pipeline and put each SET value into its $set stage as it was. In a pipeline that value is an expression:

• A string that starts with $ is a field path or a variable. {"S": "$item_data"} stored the item_data document in s.S, which no longer converts back to an item (S value must be a string in the server log), and $$NOW stored a date. A string set, a list or a map value holding such a string did the same, and a map member named $literal stored {"S": "v"} as the map, because MongoDB read {"$literal": {"S": "v"}} as the operator. The update answered InternalServerError, every later GetItem of the item did too, and so did every Query of its partition and every Scan of its table, because one unreadable item fails the page.
• $$REMOVE left s as an empty document.
• $, a map member named $k or $add, and a map member whose name holds . made MongoDB refuse the update, which answered InternalServerError and left the item unchanged.
• A document value merges into the embedded document it replaces. A SET of a map over a map kept the old members and answered 200, and a SET that changed the attribute's type (S to N, L to M, M to L, SS to S) stored two type tags and left the item unreadable.

A SET without a numeric ADD uses an operator update, where values are literal, and was not affected.

The change

Each SET value in the pipeline stage is wrapped in $literal, which makes it a constant. The fix is one line in data_engine.rs, plus the comment beside it. try_build_native_update reads no field of the engine and is now an associated function, so a unit test calls it without a MongoDB connection.

What the service answers

Measured in us-east-1, each case twice: with SET #s = :s ADD #n :one on 2026-10-25 at 23:15 UTC, and without the ADD, over existing attributes and with REMOVE on 2026-10-26 at 00:55 UTC. The service stores every SET value as sent and replaces the attribute it overwrites, with or without the ADD: $item_data, $$ROOT, $, $$REMOVE, the text {"$literal": 1}, a string set and a list holding $ strings, a map whose value is $item_data.pk, maps with members named $k, $literal, a.b and $c.d, and N, BOOL, NULL and B values. A REMOVE in the same update as the ADD removes its attribute.

Tests

tests/test_update_item_dollar_values.py: 41 dual-target wire tests. The 16 values with and without the ADD, four SETs over an attribute the item holds (map over map, nested map over nested map, a type change from S to N and from L to M) with and without the ADD, and a REMOVE in the same update as the ADD. One of the 16 values is {"M": {"a.b": {"S": "v"}}}, a dotted member name in a value that holds no $. Each test uses a table of its own and writes an item right before its update, so on MongoDB the update runs on the native path.

crates/storage-mongodb/src/data_engine.rs: a unit test parses SET #s = :dollar, #p = :plain ADD #n :one with :dollar = {"S": "$item_data"} and :plain = {"S": "x"}, builds the update and checks that the pipeline's $set stage holds {"$literal": {"S": "$item_data"}} under item_data.s and {"$literal": {"S": "x"}} under item_data.p. The wire tests check the stored value, and the session-scoped path stores it as sent too, so they would pass if the update stopped taking the pipeline. The unit test fails when the stage holds the bare value, and when only values whose text starts with $ are wrapped.

Run locally with the first two commits, when the new file held 39 tests, against MongoDB 7 (single-node replica set on a loopback port):

• the new file on main: 15 failed, 24 passed. The 15 are the updates with the ADD whose value holds a $ string, a $ or dotted member name, or overwrites an existing attribute, plus the REMOVE with the ADD; every update without the ADD passes on main.
• the new file at the head: 39 passed.
• at the head, the new file with the UpdateItem, update expression, atomic counter and RFC-0003 concurrency files under tests/: 91 passed, 1 skipped.
• 64 writers sending 1,280 SET #s = :dollar ADD #n :one calls to one key end with n = 1280 and s as sent, every call on the pipeline path.
• SQLite: the new file 39 passed at the head.
• Altered versions: wrapping only values that hold a $ fails the four SET-over-existing tests with the ADD, and wrapping the $$REMOVE of a REMOVE fails the REMOVE test.
• cargo +1.97.0 fmt --all -- --check, cargo +1.97.0 clippy --all-targets -- -D warnings, cargo test --workspace on stable and on 1.88.0 (1,767 passed each), and the MongoDB --comprehensive suite (331 passed).

Run locally on 2026-10-28 at the head, against MongoDB 7.0.45 (single-node replica set on a loopback port) and SQLite:

• the new file: 41 passed on MongoDB and 41 passed on SQLite.
• the new file with the fix commit reverted: 16 failed, 25 passed on MongoDB. The 16 are the 15 above and the map whose only member name holds ., with the ADD. MongoDB refuses that stage with error 40183 (cannot use dotted field name 'a.b' in a sub object) and the update answers InternalServerError.
• Altered version: wrapping only the values that hold a $ fails 5 tests, the four SET-over-existing tests with the ADD and that map with the ADD. m_dotted_member_names, which also holds $c.d and $e, passes under it.
• cargo +1.97.0 test -p extenddb-storage-mongodb --lib: 123 passed, 22 ignored. With the stage holding the bare value, the new unit test fails.
• cargo +1.97.0 fmt --all -- --check, cargo +1.97.0 clippy -p extenddb-storage-mongodb --all-targets -- -D warnings and cargo +1.88.0 check -p extenddb-storage-mongodb --all-targets.

CI runs the new file in run-integration, run-integration-sqlite, mongodb-pytest and cassandra-pytest.

Not changed here

When SET or REMOVE and ADD in one update name the same attribute, the service answers ValidationException Two document paths overlap. ExtendDB does not refuse it: MongoDB answers InternalServerError and SQLite applies both. That is on main and unchanged.

Files

crates/storage-mongodb/src/data_engine.rs is 4,431 lines on main and grows by 26: three for the comment, 24 for the unit test, and one fewer for the &self parameter. Of the 14 other open PRs that change it, 13 merge clean with this branch (checked on 2026-10-28). #473 conflicts in one hunk: it adds a condition on the line above the try_build_native_update call, which this branch changes to Self::try_build_native_update, and the resolution keeps both lines.

Corrected text for the fix commit's message

The squash or merge message is filled from the commit messages; the commit on the branch is not rewritten. Its first paragraph and first bullet read:

An unconditional UpdateItem that asks for no old values and no consumed capacity, on a MongoDB table with no index and no stream, is built as a native update while the server records that the table has no index. The first write on the session-scoped read-modify-write path that finds no such record, or one older than 60 seconds, reads the index catalog and records the answer; no write renews the record during its 60 seconds. With a numeric ADD it is an aggregation pipeline, and each SET value went into its $set stage as it was. In a pipeline that value is an expression:

• A string that starts with $ is a field path or a variable, so SET #s = :dollar ADD #n :one with :dollar = {"S": "$item_data"} stored the item_data document in s.S; the update and every later read of the item answered InternalServerError, and so did every Query of its partition and every Scan of its table. $$NOW stored a date, $$REMOVE left s empty, and a map member named $literal stored {"S": "v"} as the map; $, a member named $k and a member whose name holds . made MongoDB refuse the update.